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(57) ABSTRACT 

A system is presented for two-way digital multimedia broad- 
cast services, enabling a variety of interactive and other 
applications. These applications include: navigation from 
one video program to another by selection of objects in the 
current video; creation and transmission of records of user 
viewing selection histories; iterative video-based data search 
and retrieval; dynamic customization of coordination 
between video content of the broadcast stream and data 
embedded therein; and the dynamic creation of "personal- 
ized" programs by the user without, through the broadcast 
and selection of overlapping program segments which are 
themselves customized for a particular user or group of 
users. 

3 Claims, 9 Drawing Sheets 



0 



MINUTES 



/ 



VIEWER SELECTS TOPIC A 



Ai 


A 2 A 3 


A 4 






A 5 . 


A 6 


Ci 


A 7 



^92 
-93 



r 



94 



30 



STREAM CONTENT MODIFIED 



MINUTES 



07/22/2004, EAST Version: 1.4.1 



U.S. Patent 



Oct. 2, 2001 



Sheet 1 of 9 



US 6,298,482 Bl 



SET-TOP BOX 



r 



INPUT 
PORT 



TUNER 



£ 



8 



OUTPUT 
PORT 



£ 4 _£! 



DEMOD 



DEMUX. 



^ AUDIO 
DECODER 



VIDEO 
DECODER 



MICROCONTROLLER 



10 



2\ 



RAM 



INFRARED 
INTERFACE 



£ 



81 



DATA 
PORT 



£ 



11 



VIEWER PROFILES 



SELECTION HISTORY 
RECORDS 



HISTORICAL REPORTS 



ON-SCREEN 
MESSAGE BITMAPS 



12 



13 



■14 



15 



FIG.1 



07/22/2004, EAST Version: 1.4.1 



U.S. Patent 



Oct. 2, 2001 Sheet 2 of 9 



US 6,298,482 Bl 



FIG.2A 
FIG.2B 



FIG.2 



r 



21 



ADD 

'Tune begin* 

RECORD 
TO SH 



z: 



24 



ADD 
"TUNE END" 
RECORD 
TO SH 



SYSTEM 


POWER -ON 


r 








SYSTEM 


INITIALIZATION 


r 




16 



•17 



vrc , TIME FOR ^ 
-^<?ERIODIC HISTORY 



DISPLAY PROGRAM MENU 



19 



USER PROGRAM SELECTION 



r 



20 



PROGRAM DISPLAY 



22 



YES 




FIG.2A 



07/22/2004, EAST Version: 1.4.1 



U.S. Patent Oct. 2, 2001 Sheet 3 of 9 US 6,298,482 Bl 



NO 




YES 



SEND REPORT TO HEAD END 



FIG.2B 



07/22/2004, EAST Version: 1.4.1 



U.S. Patent 



Oct. 2, 2001 



Sheet 4 of 9 



US 6,298,482 Bl 



37 



HEAD END 
MODIFIES 
DATA OR VIDEO 
CONTENT OF 
PROGRAM 



36 



HEAD END 
PERFORMS 
KEYWORD 
SEARCH 
(OPTIONAL) 



34 



16 
17 
19> 





SYSTEM 


POWER— ON 








SYSTEM 


INITIALIZATION 



DISPLAY PROGRAM MENU 



20 



USER PROGRAM SELECTION 



32 



PROGRAM SENT TO USER 



22 



i 



PROGRAM DISPLAY | 




^-33 




SEND 




REPORT 




TO HEAD 




END 



MESSAGE 
DISPLAY 



USER YES OR 
Y £S ^RESPONDS TcN N0 
^HYPERLINK. 
? 



IF DATA SELECTED, DELIVER 
DATA TO USER SYSTEM 



35 



ADD "HYPERLINK" 
RECORD TO SH 



FIG.3 



07/22/2004, EAST Version: 1.4.1 



U.S. Patent Oct. 2, 2001 Sheet 5 of 9 US 6,298,482 Bl 



38^ 

39- v 

40- ^ 



RECORD NUMBER 



DATE (YY/MM/DD/) 



time (HH:MM:SS) 



41- 



VIEWER PROFILE NAME 



42 



ACTION: 
TUNE BEGIN 



TUNE END 



VIDEO HYPERLINK 
SELECTION 



INTERACTIVITY 
SELECTION 



43^ 


CHANNEL 


44-^ 


PROGRAM 


NAME 


45-^ 


PROGRAM 


CATEGORY 



46^ 
47^ 
48^, 



NAME OF 

SELECTED OBJECT 



SUB-SELECTION 
INFO. 



STORAGE/DISPLAY 



49 



INTERACTIVITY 
APPLICATION TYPE 



SELECTION 
INFO. 



50 



fig.4 



07/22/2004, EAST Version: 1.4.1 



U.S. Patent 



Oct. 2, 2001 



Sheet 6 of 9 



US 6,298,482 Bl 



51 A PROFILE NAME 



52 



VIEWER NAME 



53 



VIEWER MAIL ADDRESS 



54 



VIEWER E-MAIL ADDRESS 



55 



VIEWER TELEPHONE NUMBER 



56 



VIEWER FAX NUMBER 



57 ^ VIEWER AGE 



58 



VIEWER SEX 



59 



HISTORICAL REPORT PERIOD 



60 



HISTORICAL REPORT TYPE 



61 



VIEWING PREFERENCES 



62 



RATING INFORMATION 



63 



INSTANT FEEDBACK ENABLE/DISABLE 



64 



PERSONAL INFORMATION ENABLE/DISABLE 



82 



MESSAGE FILTERING INFORMATION 



FIG.5 



07/22/2004, EAST Version: 1.4.1 



U.S. Patent Oct. 2, 2001 Sheet 7 of 9 US 6,298,482 



PROFILE NAME 



START DATE 



END DATE 



AVERAGE NUMBER OF HOURS VIEWED PER DAY 



MINIMUM NUMBER OF HOURS VIEWED PER DAY 



MAXIMUM NUMBER OF HOURS VIEWED PER DAY 



NUMBER OF HYPERLINKS SELECTED 



NUMBER OF INTERACTIVE APPLICATIONS SELECTED 



NUMBER OF PROGRAMS VIEWED 



NUMBER OF COMMERCIALS VIEWED 



VIEWING PREFERENCES OVER SUMMARY PERIOD 



FIG.6 



07/22/2004, EAST Version: 1.4.1 



65-^ 
66^ 



67 



68^ 
69^ 
70^ 

71 ~N 

72 



73 



74 



75 



U.S. Patent Oct. 2, 2001 Sheet 8 of 9 US 6,298,482 



PROFILE NAME 



START DATE 



END DATE 



NUMBER OF SELECTION HISTORY RECORDS INCLUDED 



SELECTION HISTORY RECORD 1 



SELECTION HISTORY RECORD n 



FIG.7 



07/22/2004, EAST Version: 1.4.1 



76^ 
79^ 



U.S. Patent 



Oct. 2, 2001 Sheet 9 of 9 



US 6,298,482 Bl 



Ai 


A 2 


A 3 


Bi 


B 2 




Ci 


C 2 


c 3 


Di 


D 2 


D 3 



91 



-10 



20 



FIG.8a 



30 



81 

82 
83 
84 



MINUTES 



Ai 


A 2 


A 3 


A 4 


Bi 




B 3 




Ci 


C 2 


c 3 


c 4 



0 



15 



FIG.8b 



30 



85 
86 
87 



MINUTES 



Ai 


A 2 




B 2 


Ci 


C 2 


c 3 


c 4 



^-88 
89 
90 



FIG.8c 

VIEWER SELECTS TOPIC A 



A 1 A 2 A 3 


A 4 




B 2 


A 5 


A 6 


Ci 


A 7 



-92 
-93 
■94 



0 mmS STREAM^CONTENT MODIFIED^" M ' NUTB 



FIG.9 



07/22/2G04, EAST Version: 1.4.1 



US 6,298 : 

1 

SYSTEM FOR TWO-WAY DIGITAL 
MULTIMEDIA BROADCAST AND 
INTERACTIVE SERVICES 

DESCRIPTION 

1. Technical Field 

This invention relates to a method of delivering data from 
a server to a plurality of subscribers connected to the server. 

2. Description of the Prior Art 

ITV systems require a network architecture which sup- 1Q 
plies an "upstream" (from user to head end) and a "down- 
stream" (from head end to user) channel to each user. Most 
commonly, the downstream channel has been the TV 
broadcast, satellite, or cable medium, while the upstream 
channel has used the public service telephone network. 5 
Typical ITV applications have included viewer polling, 
home shopping, and distance learning. These applications 
were frequently menu-based, and static — i.e. a modest num- 
ber of choices in a given category were pre-programmed 
into the system and offered to users. 

In U.S. Pat. No. 4,536,791, a method is presented for 
"channel monitoring" — i.e. the transmission to the head end 
of information regarding current viewing information, such 
as when a set is turned on, what channel it is tuned to, and 
when the channel is changed. This patent also presents an ^ 
"interactive information retrieval" capability, with which the 
viewer may access a data bank by interacting with the head 
end by use of a non- video TV channel (uses keyboard or 
keyboard-like remote.) U.S. Pat. No. 4,924,303 describes a 
system which allows users to search for and retrieve still 30 
frame video images and audio segments. U.S. Pat. No. 
5,355,480 describes an interactive TV terminal for user with 
services such as in-room hotel checkout. A distance learning 
ITV system is described in U.S. Pat. No. 5,537,141. This 
system supplies prepared pre-recorded audio responses 35 
which are customized for individual students' responses to 
questions. 

U.S. Pat. No. 5,550,578 describes an ITV system for the 
delivery of video and data, the latter from a variety of 
information services. 40 

Hie A. C. Nielsen company (Nielsen Media Research 
Interactive Services Co.) utilizes special-purpose hardware 
(the "Storage Instantaneous Audimeter" and "Nielsen 
People Me ter") for the purpose of monitoring TV viewing 
behavior. These devices monitor home viewing per house- 45 
hold member, and report to a central location by use of 
telephone lines. Because of the complications involved in 
distributing this hardware, only a limited number of these 
units are in use (5000 units, as of July 1997, distributed to 
"Nielsen families"). The system presented here requires no 50 
special purpose hardware, allowing a larger number of 
viewers to be included in the sample. Previous monitoring 
systems were limited to the use of the phone line in both the 
head-end-to-user direction and vice versa, causing compli- 
cations if another device (e.g. a fax machine) answered the 55 
phone, receiving the message from the head end. The system 
presented here gives a dedicated channel to the head end for 
control of the STB, namely the private data channel in the 
digital broadcast stream, as described below. In addition, the 
system presented here allows a finer granularity of viewer eo 
selection report to the head-end (i.e. a single program 
selection or termination) and a shorter reporting period (an 
immediate report following the selection — i.e. "instant feed- 
back" as described below — as opposed to a daily viewing 
report.) 65 

Patent applications, Ser. Nos. 08/854,227 and 08/854,225, 
assigned to the same assignee as that of the instant 
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application, describe systems, which customize the program 
material which is sent to the user. For example, the (video) 
commercials seen by the viewer of a video program are 
customized based on the viewer's interests. 

The above mentioned systems customize content without 
the user's participation, on the basis of historical profiles, 
not allowing user "navigation" through databases, as 
described below. 

There is, therefore, a need for a dynamic system for the 
interactive delivery of data. By "dynamic", we mean rapidly 
adaptive. More specifically there is a need for a system 
which: 

1. enables the user to navigate through a vast collection of 
resources, such as a multimedia database (as opposed 
to a small number of pre-defined selections); 

2. enables this navigation to be performed within the 
medium itself, e.g. navigation (switching to another 
video program) through a group of videos by interac- 
tion with the objects in videos themselves, as opposed 
to navigation by use of another medium, e.g. searching 
through supplementary information such as a text list of 
video titles and keywords; 

3. delivers multimedia resources to the user on the basis 
of selection and search activity performed by the user 
and the head end in tandem. 

4. spontaneously modifies both the content of the digital 
stream (e.g. video and data files) as well as the asso- 
ciation of objects in the stream (e.g. the association of 
objects appearing in the video to text files on related 
topics.) 

BRIEF DESCRIPTION OF THE DRAWINGS 

FIG. 1 depicts the elements required in a set-top box 
(STB) which is capable of implementing the VRS. 

FIG. 2 is a flowchart showing the basic operation of VRS 
in the context of regular (video) program viewing. Also 
shown are the operation of VRS when multimedia data is 
embedded in the program, in which case information avail- 
ability messages are filtered by the user's viewer profile. 

FIG. 3 is a flowchart displaying the operation of VRS for 
iterative user/server data search and retrieval. The main 
feedback loop shown in the figure demonstrates the use of 
VRS reports, sent from the STB to the head end, in itera- 
tively refining the embedded data sent to the user within the 
multiplex which is sent to the viewer (see below for 
definition). 

FIGS. 4 through 7 show the format of various data 
structures which are used in implementing the VRS. FIG. 4 
shows the format for the selection history record. 

FIG. 5 shows the format for the viewer profile informa- 
tion. As described in detail below, a profile, containing 
general information concerning the viewer's preferences, is 
chosen at the start of each viewing session. During the 
session, the subscriber selection history information is added 
to the selection history associated with that profile. 

FIG. 6 shows the format for the "summary" type of the 
VRS historical report. 

FIG. 7 shows the format for the "detailed" type of the 
VRS historical report. 

FIG. 8 shows how the overlapping program segments in 
the stream are scheduled, in a synchronous (FIG. 8a), 
semi-synchronous (FIG. 86), or asynchronous (FIG. 8c) 
manner. The horizontal axis indicates time, while the verti- 
cal axis indicates three or four media streams (see Summary 
for definition), labeled A, B, C, and D. 
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FIG. 9 shows an example of how the content of the digital first video program is transmitted from the head end to the 

stream can be modified at the direction of the user or group subscriber. An object in the first video program can be 

of users. Originally, three overlapping media streams, A, B, selected, where the object is an image in the first video 

and C are transmitted. When the user (or group) indicates program and where the object is not part of any supplemen- 

interest in the topic depicted in segment A^, the video (or 5 tary information added to the first video program. This 

audio or data) content of the stream is modified at the head selection information is then transmitted to the head end. In 

end, and all subsequent program segments are related to this response to the received selection information the head end 

topic. then transmits a second video program to the subscriber, 

where the second video program corresponds to the object 

SUMMARY OF THE INVENTION 10 sc lected in the first video program. 

The invention is a viewer response system (VRS) for the In ? ct ai J oth « . of ^is invention a method of 

two-way digital multimedia broadcast environment. It acquinng subscriber selection history information on a num- 

enables many of the applications associated with interactive ber of subscribers is provided. With this method selected 

television (ITV), plus many new applications (described V1 ^ e0 Programs are transmitted from the head end to the 

below) which are not enabled with existing ITV systems. 15 subscribers m response to requests for the selected video 

Ti . i_. . -j . u- programs from the subscribers. Each of the subscribers 

It is an object of this invention to provide a system which i * * ■ • j • j l l 

- * 4 - ■ . i • selects objects in a received video programs, where each 

performs statistic gathering on viewer selections in a manner . . . . c • * * r 

u* u • u* ■ * .u j * ■ i ■ _ 4 . i object is an image in one of video programs, is not a part of 

which is unobtrusive to the user and which is automatic and J ljjjj. j j 

. t ^ i i . . - 4 . any embedded data added to the programs, and corresponds 

instantaneous, at the discretion of the user. „„ t £t t . . . r *• ■ • . i_ • i_ 

' 20 to a file containing information on a subject which corre- 

It is another object of this invention to provide a system sponds to the appearance of each respective object. Finally, 

in which VRS-capable STB's can be mixed with other a sclcctiori database history of the object selections is 

STB's with no adverse effects on each other, or complica- maintained. This selection database history can then be used 

tions for the system. to dynamically adjust the correspondence of the objects to 

It is another object of this invention to provide a system 25 the files, where each subscriber receives customized infor- 

in which VRS-capable STB's can be used in networks mation in response to selecting an object, 
without VRS capabilities, and perform conventional STB 

functions. DESCRIPTION OF THE PREFERRED 

In the present invention, the head end broadcasts a mul- EMBODIMENT 

tiplexed stream to viewers connected to a shared medium. 30 present invention assumes the availability of two 

This stream contains video and audio content as well as communication paths between the head end and the viewer 

control information and embedded data (see Description of (or user.) The path from the head end to the viewer (known 

the Preferred Embodiment.) ^ tne "downstream" path or channel) is, of necessity, of 

The new applications enabled by VRS include: higher bandwidth than the path from the viewer to the head 

Intra- Video Navigation— i.e. the ability for the viewer to 35 end (the "upstream" path or channel.) The downstream path 

move from one video (and audio) program to another is used to deliver the video, audio, data, and other multi- 

without exiting the video display interface; media content to the viewer, as well as control information. 

\tewer Response Monitoring— Keeps track of viewer ^ upstream path is used to deliver viewer requests and 

selection information and periodically reports it to the selections to the head end. 

head end, if enabled; The digital stream which is transmitted on the down- 
Iterative Video-Based Data Search and Retrieval— stream path is a multiplex of streams of various media, such 
Allows the viewer of a program to select and modify ^ ^0 and audio, and may contain multiple streams of 
the data which is embedded in a program program each media tv P e - ^ example is a video program with 
dynamically, while viewing, in order to search and 45 several associated audio streams for different languages. The 
retrieve items from a multimedia database; media steams may be coordinated, such as the video and 
Dynamic Customization of Embedded Data- each audio stream in this example, or uncoordinated, such as 
Spontaneously adapts to viewers' current interests by ^ dls P f ate au *° streams - ^ media streams ma y or ma V 
modifying data files embedded in the stream and their not overla P m tune - 

correspondence with video objects; 50 ^ P rcsent invention requires the use, at the viewers' 

Personalized Program Creation-By broadcasting over- Pre™, °* a set-top box (STB). The STB receives the 

lapping video (and audio) program segments, enables f* mm * downstream signal, tunes to the desired channel, 

the creation, by the viewer (or automatically by the demultiplexes and decodes the desired media stream(s), 

viewers* STB) of a display program suited to the filtc L rs out ™wantod information, and controls the interface 

viewers' interests 55 to me ncac * en " via me u P stream channel. The STB may or 

Accordingly, this invention is a method of customizing ma ? not havc stora § c capabilities. It is described below in 

video programs received by each subscriber by a group of morc dctaiJ * 

subscribers. With this invention a set of program segments ^ following terms are used in the discussion that 

of the program are transmitted from the head end to the follows: 

group of subscribers, where the set is selected by the head eo A media stream is a digital bitstream of a specific 

end server in accordance with profile information on group medium, such as a video bitstream for a one-hour movie, 

of subscribers. Then each of the subscribers selects a respec- FIGS. 8a-8c shows media streams 81-90 and FIG. 9 shows 

live ordered subset of the set of program segments. In one media streams 92-94. 

specific aspect of this invention subset can be ordered by the A program segment is a section of a media stream, whose 

subscriber without sending any data to the head end server. 65 duration is shorter than or equal to the duration of the media 

In another aspect of this invention, a method of navigating stream. An example is a five-minute video program segment 

through a video database is provided. With this method a in a ten-minute video media stream. Program segments are 
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generally self-contained. FIG. Sa shows program segment 
Dj 91 as well as A^Aj, Bj-B^ ^-C^ and D 2 and D 3 . 

A multiplex is a set of media streams which are coordi- 
nated for concurrent play, such as a video media stream and 
an audio media stream. 

The term program (as used here) refers to the continuous 
presentation of material selected for viewing (and listening) 
by the viewer. An example is a news program, consisting of 
the continuous presentation of audio and video material as 
selected by the viewer. Embedded data can also be included 
in a program. 

Objects are well-defined sub-units discernible in a media 
stream, such as visible objects ("video objects") in a video 
stream, words ("audio objects") in an audio stream, or files 
("data objects") in a data stream. In the following 
discussion, "data" is used to refer to various types of 
information, such as text files, computer programs, pointers 
to various media types, messages to be displayed on-screen, 
and control information. "Embedded data" can include all of 2 q 
the above types of information, as well as multimedia 
information such as video and audio content. 
Intra- Video Navigation 

The present invention enables the provision, to the user, 
of the capability to navigate through a large number of video 
(and other multimedia) resources in a unique fashion. The 
user can move within and between video sources without 
exiting the viewing of the video display. By selecting video 
objects (see above) within the current video, the viewer is 
given the ability to move to one or mare alternative video 
sources. From that video, the same capability can move the 
user to another video source, and so on. 

This capability is enabled by preparing the source video 
by marking the location of significant video objects in the 
video content. The same "marking" can be done for audio 
objects. In the marking process, the objects are uniquely 
identified and flagged so that the user, in playing the source, 
will be aware of the presence of a marked object. The 
marking also identifies other video, audio, or data objects in 
the stream which are related to the marked object, and, or 
pointers to related objects which are not currently in the 
stream. An example of a marked object is the image of a 
person which appears in a set of video frames. 

In selecting a marked object (using a user interface such 
as that which is described in the preferred embodiment) the 
user is presented with the alternative media choices associ- 
ated with the selected object. The association of video (or 
audio) object and alternative media choices is flexible and 
dynamic, as described in "Customization of Embedded 
Data" below. The alternative may be a video source, audio 
source, data file, etc. In the context of intra-video navigation, 
it is an alternative video source. 

Once the user has selected from among the alternatives 
which are displayed, the STB relays this selection to the 
head end, in addition to storing the selection information in 
the user's selection history (see "Viewer Response 
Monitoring"). In response to the selection, the head end 
modifies the video (and/or audio) stream within the multi- 
plex which is sent on the downstream path. It does so by 
switching from the current video (and/or audio) media 
stream to a new media stream, as indicated by the viewer's 
selection. 

The above discussion assumes that the user has a dedi- 
cated video program to himself. If this is not the case, similar 
navigation can be done on the basis of the consensus of 
groups of users, whose selections are used to poll user 
interest in moving to another program. 
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Viewer Response Monitoring 

VRS enables a variety of functions intended for keeping 
track of a user's viewing history. This historical information 
is used, both at the head end and at the STB, for customi- 
zation of content which is delivered to the user. In addition, 
this information is used by the head end to monitor the 
popularity of programs and advertisements, and to assemble 
demographic information on viewers. 

VRS enables these functions (see FIG. 1) by use of a 
microcontroller 9 and RAM 11 in the user's STB 1. This 
STB contains the typical elements for the reception, 
processing, and decoding of digital multimedia broadcasts. 
The incoming stream, sent from the head end over a 
network, arrives via an input port 2, and a tuner 3 selects the 
desired channel. A demodulator 4 extracts the digital stream 
from its analog, modulated format. A demultiplexer 5 filters 
out the selected program in the digital stream, which is then 
decoded by video 6 and audio 7 decoders. An infrared 
interface 10 decoder receives selection commands from the 
user for channel tuning, interactivity, and hyperlink com- 
mands (see below.) In order to reduce the amount of RAM 
required in the VRS STB, an optional data port 81 may be 
included. Using this serial or parallel port, users can connect 
the STB to a personal computer (PC) for storage and 
retrieval of large files. 

When VRS is installed on the user's premises equipment 
(and optionally when other viewers wish to use the system), 
a viewer profile 12 is created for the user and stored in the 
STB RAM. The format for this profile is shown in FIG. 6 
and will be discussed below. The profile is created by use of 
an on-screen menu display, similar to those used for adjust- 
ing picture controls, using the infrared remote control unit. 
Since a STB may be shared among several users (such as 
family members), each STB can support a large number of 
profiles, and a profile is selected during the initialization 
process. There may be multiple profiles per user, if so 
desired. 

Each time the user's system is powered on, 16, after 
system initialization 17 (which includes identification of the 
viewer profile to be used for this viewing session), the 
microcontroller determines if a periodic historical report is 
due 18, based on the "historical report period" parameter 59 
in the viewer profile and the current time and date. This 
parameter is adjustable by the user. If a report is due, it is 
sent to the head end via the STB output port 8. The report 
may be a summary of viewing activity over the report 
period, or a more detailed description of viewing behavior, 
based on the "historical report type" parameter 60 in the 
viewer profile. The "personal information enable,idisable" 
parameter 64 in the viewer profile information allows the 
user to control the amount of his personal information which 
is included in the historical report. 

VRS historical reports containing subscriber selection 
history information can be used by the head end to monitor 
the popularity of programs and commercial messages. In 
addition (assuming a short report period such as a few 
hours), the VRS reports may be used by the head end to 
determine the status of the user's equipment and to diagnose 
problems. Reports to the head end can also be prompted by 
the head end by use of a "send report" message sent by the 
head end in the "private data" area of a video stream or in 
a non-video control stream. The collection by the head end 
of historical reports can also be done at times when the user 
is not viewing programs, such as during late night hours, 
provided the STB is powered on. 

The program menu 19 is displayed to the user to notify 
him of the viewing options which are available. The user 
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selects a program for viewing 20 which then beings display have any number of viewing profiles, if so desired. This may 

22. At the same time, an entry is added to the "selection be useful in offering a variety of viewing session types to the 

history" (SH) 13 in the STB RAM to record this program user, for example, "academic" viewing, where embedded 

selection. The entry indicates "tune beginer" i.e. the user has data is critical, versus "entertainment" viewing, where it is 

tuned to a program. The format for the selection history 5 less so. These profiles are differentiated by profile name 51. 

record (SHR) is shown in FIG. 4 and will be discussed Rating information 62 allows the head end and STB to know 

below. If the "instant feedback enable disable" parameter in what material is appropriate for this user and can be used to 

the viewer profile 63 is set to "enable", a message indicating prevent the transmission or display of offensive material, 

this program choice is sent to the head end via the STB When a user completes the process of creating a new viewer 

output port. Since the head end collects more data from 10 profile, it is stored in the STB's non-volatile RAM and a 

those users who have enabled instant feedback, incentives copy is sent to the head end. 

may be offered to these viewers to encourage them to enable Iterative Video-Based Data Search and Retrieval 

this function, and to enable the transmission of personal FIG. 3 shows the operation of the VRS when used for the 
information from the viewer profile. Thus, the granularity of function of iterative video-based user/server data search and 

the message sent to the head end is a single viewer selection, 15 retrieval. 

VRS thus allows the head end to gather statistics on viewing Following system is power-on, the system initializes and 

selections by the user without requiring any effort from the offers a program menu to the user. The user selects a 

user, and without allowing tampering of this information by program for viewing from those offered. The program 

the user. It is also unobtrusive to the viewing process. selected, in this case, contains embedded data — i.e. data (or 

Viewing of the program continues until the viewer ter- 20 other multimedia) content related to the topics displayed in 

minates the viewing by ending the viewing session (i.e. the program. Messages pertaining to the embedded data may 

turning off the system) or tuning to a different program. In or may not be filtered out by the microcontroller, depending 

either case, a "tune end" SHR 24 is added to the SH and sent on the viewer's history and interests. The embedded data is 

to the head end, if instant feedback is enabled. offered to the viewer by display of a "hyperlink". When the 

If the program being viewed has embedded data 25 the 25 user selects a hyperlink for the delivery of a unit of embed- 

microcontroller, on the basis of the "message filtering data" ded data, a selection history record (SHR) is written to the 

82 in the viewer profile will determine if the message selection history (SH) 15. With the delivery of the selected 

announcing the availability of the embedded data should be embedded data to the user 34, a report indicating the 

shown to the user 26. If so, it is displayed on screen 31, using selection of the embedded data is sent to the head end (if the 

the on-screen message bitmaps 15 which are stored in the 30 viewer has enabled this function.) Upon receipt of this 

STB RAM. Alternatively, the message can consist of an report, the head end system optionally performs a database 

audio tone or "pre-recorded" audio phrase, e.g. "embedded search 36 on the basis of keywords from the report, 

data is now available". Optionally, the user could be prompted to supply additional 

If the user responds to the availability message (or to keywords, 

information in the program regarding interactivity options 35 Using the search results (or the contents of the report, if 

such as shopping or polling), a "hyperlink" or "interactivity" a search was not performed), the head end will alter the 

record is added to the SH and sent to the head end, if instant content of the embedded data to match the current interests 

feedback is enabled. of the user and the "zero in" on specific topic areas in which 

The selection history records (see FIG. 4) are used both by he has shown interest. Unless the user has his own dedicated 
the user's system and the head end as records of individual 40 video channel (i.e. Video-on-Demaod), the head end's con- 
viewing decisions by the user. They are used in the creation tent alteration will be on the basis of responses from groups 
of "summary"-type VRS historical reports (see FIG. 6) and of users. In effect, this creates a "polling" system, whereby 
"detailed" -type reports (see FIG. 7). A SHR for a "tune the group of users lets its interests be known to the head end, 
begin" event records, in addition to the profile name, date which responds to the consensus of a given percentage of the 
and time, the channel tuned to, the program name, and the 45 users. 

program category (e.g. Dews, drama, etc.) A SHR for a "tune The user(s) respond(s) to the new embedded data based 

end" event records only the profile name, time, and date. For on his interests, causing another alteration by the head end 

video hyperlink selection, the SHR records the name of the of the embedded data, and so forth. Thus the search and 

object selected, sub-selection information (e.g. when select- retrieval of data is an iterative adaptive process engaged in 

ing a person in a video, the user may choose between a short 50 by the user(s) and the head end in tandem, 

biography, long biography, or video clip relating to this Dynamic Customization of Embedded Data 

person), and whether the user stored the related information As described above (see "Intra- Video Navigation"), 

or displayed it immediately. For an interactivity selection, objects in the multiplex are marked in preparation for the 

the SHR records the type of interactivity application (e.g. transmission to users of the program. In the marking 

home shopping) and the selection information. 55 process, an object, such as a video object, is associated or 

Detailed VRS historical reports contain a number n of corresponds with related multimedia objects in the stream 

SHR's 80 in addition to the basic profile information and and/or pointers to objects which are not currently in the 

report start and end dates. Summary-type VRS historical stream. The association information is itself sent in the 

reports summarize the viewing behavior of the user with this stream, as control information, and is monitored by the STB. 

profile over the report period, including number of programs 60 The head end can dynamically modify both the content of 

viewed 73, average number of hours viewed per day 68, and the stream and the control information associating objects in 

viewing preferences 75. the stream, in response to user interest. If a user (or group of 

The viewer profile (see FIG. 5) contains viewer profile users) demonstrates, by selection of embedded data and/or 

information and is meant to summarize the requirements and intra-video navigation, interest in a particular topic, the data 

preferences of the viewer, for use by the user's system and 65 embedded in the stream can be modified to reflect this 

the head end. It can be transmitted to the head end for the interest. This modification can occur during the viewing of 

latter's use, during the process of profile creation. A user can the program, within seconds of the indication, by the user (or 
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group of users) of a new topic of interest. The control Alternatively, the program segment selection may be done 

information must be modified as well, to establish the for the user automatically, by the STB. Using the selection 

association of the new embedded data with the previously history summary information, the STB can select among the 

marked objects in the video. available program segments, provided they are associated 

In addition to the above, the head end can dynamically 5 with keywords identifying their topic areas (to enable the 

modify the associations alone, without modifying the data STB to choose among them.) 

content. For example, the data embedded in the stream of a For the example of FIG. 8a, a user (or STB) may select 

sports program may include historical, statistical, and anec- the segment set (B l9 D 2 , and A$). In this case, there is no 

dotal information related to the players (i.e. the video display overlap, and no additional STB storage is required, 
objects). If a majority of viewers show an interest in 10 The STB then demultiplexes the desired set of program 

statistical information when selecting the embedded data segments in sequence to display the personalized program, 

associated with a player, this information could be displayed Segments may also be stored for future viewing, in case the 

immediately on screen when a player is selected. viewer selects too many segments for display in the program 

Alternatively, a menu offering statistical, historical, or anec- time. Segments may also be re-ordered, if necessary or 
dotal information could be displayed when a player is 15 desirable, such as in the case where a user selects segments 

selected. which overlap entirely. For example, in the stream depicted 

FIG. 9 shows an example of the dynamic modification of in FIG. 8a, a user may select segments D lf B l7 and A u and 

stream content in response to a user selection. request that they be played in that sequence. Re-ordering 

Personalized Program Creation requires STB storage for delayed play. 

In the multiplex sent to the user, multiple program seg- 20 Because of the high bitrate requirements of digital video, 

ments (see definition above) may be sent. These segments a compression method is usually applied to video before 

may overlap in time, as shown in FIG. 8. The transmission transmission over a network. In the preferred embodiment, 

of multiple program segments such as those shown in FIGS. video (and audio) content are compressed using MPEG-2 

8a-c enables the creation, by the user or automatically by (Motion Pictures Experts Group) compression, as specified 

the STB, of personalized programs based on the users 1 25 in ISO/IEC 13818-2 for video and ISO/EC 13818-3 for 

interests. audio. Typical bitrates for compressed MPEG-2 video 

Program segments may be related to different topics (such streams range from 3 to 15 Mbps. 

as different news items in a news broadcast) or the same The MPEG-2 standard also specifies how programs 

topic at different depths (such as a 5-, 10-, or 15-mihute (consisting of audio and video elementary streams) can be 

news item about a given topic.) In FIG. 8a, for example, four 30 multiplexed together in a "transport stream". This is speci- 

topic areas, A-D, are shown, and are transmitted in four fied in the MIPEG-2 Systems Specification, ISO/IEC 13818- 

overlapping media streams of the video stream in the 1. The MPEG-2 Systems Specification accommodates the 

multiplex. inclusion in a program's transport stream of non- video and 

It should be noted that the creation of these personalized non-audio streams, by use of "private data" streams. All 

programs as depicted in FIGS. 8a-£c can be done in the 35 transport stream packets, regardless of content, are of a 

broadcast mode, and does not require an upstream channel. uniform size (188 bytes) and format. "Program-Specific 

FIG. 9 shows how the upstream channel can be used for an Information", which is also carried in the transport stream, 

enhanced program creation process, one which involves carries the information regarding which elementary streams 

interactivity. have been multiplexed in the transport stream, what type of 

As shown in FIG. 8, the program segments can be 40 content they carry, and how they may be demultiplexed, 
synchronous (FIG. 8a), semi-synchronous (FIG. 8ft), or In the downstream direction, a shared broadcast medium 
asynchronous (FIG. 8c). Synchronous program segments, is used for the preferred embodiment, such as the cable TV 
for each media stream begin and end at the same points. infrastructure. Modulation methods to be used include QAM 
Semi-synchronous segments have some common juncture ' (quadrature amplitude modulation) and VSB (vestigial side- 
points (such as at the 15 -minute point in FIG. 8ft), while 45 band). Chipsets for these demodulation types are widely 
asynchronous segments have none. available and inexpensive. The bandwidth required in this 

In the case of semi-synchronous or asynchronous program direction for the implementation of VRS is at most 10 

segments, the user commencing the viewing of a program Kbits/s. 

segment already in progress has the option of viewing it In the upstream direction, the preferred embodiment of 

from the point where he entered (i.e. truncating the content 50 VRS uses the PSTN (public service telephone network). In 

up to that point) or storing the content of that segment prior conjunction with an ADSL, (asymmetric digital subscriber 

to viewing and viewing it from the start when the user is line) chipset for modulation in this direction, the user's 

ready. The latter approach requires a considerable storage home phone line may be used simultaneously for VRS 

capacity in the STB (see the preferred embodiment), as the messages and for regular voice phone service. This chipset 

storage requirement will be cumulative with each "in- 55 would be included in the output port module shown in FIG. 

progress" segment which the user joins. 1. The required bandwidth in this direction is at most 10 

Tne user interfaces with the STB at the beginning of a Kbits/s. 

program to perform the program creation. The STB receives, Tne use of VRS -capable STB's in a broadcast distribution 

from the downstream channel, control information regarding network does not interfere with the use of conventional 

the program segments available for the currently-broadcast 60 STB's in the same network. For purposes of STB polling 

program, and offers a menu of choices to the user. From (see above), the head end would need to be aware of the 

these choices, the user selects a set of program segments, address of the VRS -cap able STB's. MPEG-2 

which may or may not overlap in the time of their broadcast demultiplexers, MPEG-2 audio decoders and MPEG-2 

for display. If there is display overlap, storage may be video decoders are now widely available. The preferred 

required, as described above. One set of program segments 65 embodiment uses these integrated circuits (IC) in the VRS- 

may be designated as the default set, if the viewer does not capable STB to perform the functions indicated in the block 

make any selection. diagram of FIG. 1. If not incorporated in the audio and video 
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decoder, an intermediate IC is necessary at the output of the 
decoders to convert from digital to analog and, in the case 
of video, encode to the desired video analog signal format 
(such as NTSC, PAL, or SECAM.) S-video output from 
these ICs is optional. The MPEG-2 video decoder used in 5 
the preferred embodiment has "on-screen display" capabili- 
ties for superimposing text and other bitmaps on the video 
screen, at a variety of intensities. 

The microcontroller for the STB requires a clock and an 
EPROM for instruction storage. In the preferred 10 
embodiment, the clock rate is 25 MHz, and the EPROM size 
is 256 KBytes. The microcontroller programs the MPEG-2 
demultiplexer to create a queue for embedded data, in 
addition to those created for audio and video packets. The 
embedded data queue is monitored by the microprocessor by 15 
polling or interrupts. When this queue is found to be 
non-empty, its contents are read by the microprocessor. If 
they are message packets, their content is converted into 
bitmaps for on-screen display, This display involves the 
microprocessor programming the video decoder as to the 20 
location, size, and other parameters of the bitmap. The 
viewer may disable the display of data availability mes- 
sages. 

Hie size of the RAM needed for a VRS -capable STB in 
the preferred embodiment is (minimally) 4 Kbytes. To 25 
reduce cost, DRAM (dynamic RAM) is used. A small 
amount (1 Kbyte) of non-volatile RAM is used for data 
(such as viewer profiles) which must be preserved when the 
STB is powered off. A battery is required for this RAM and 
for maintaining the time-of-day timer. 30 

To save RAM data in anticipation of power-off, to save 
files which are larger than the RAM size, or to accumulate 
a larger number of SHR's than the RAM can accommodate, 
the STB's data port is used to transfer data to a PC. This port 
can also be used to transfer large historical reports from the 35 
PC to the STB, en route to the head end. 

The data port used in the preferred embodiment is an 
RS-232 parallel port. 

In the preferred embodiment, the audio signal is encoded 
and decoded as two channels (with the additional option of 40 
Dolby noise reduction and surround sound) and requires 
multiple speakers at the STB output. 

To indicate the presence of marked objects, video high- 
lighting is used in the preferred embodiment. This highlight- 
ing is performed by the decoder's on-screen display capa- 45 
bility at the instruction of the microcontroller, which obtains 
the coordinates of the video object from the control infor- 
mation. For audio "highlighting**, in the preferred 
embodiment, a tone is used, preceding by a short time the 
marked words. Both audio and video highlighting can be 50 
disabled by the user who is not interested in their use. 

Selection of marked objects, in the preferred embodiment, 
is performed by the use of the IR interface of the remote 
control. This interface moves an on-screen cursor to the 
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objects, and clicking on a pre -assigned button of the remote 
control is used to select video objects and objects displayed 
in on-screen menus. For the selection of audio objects, an 
"audio-related information" icon is displayed at the edge of 
the screen, which, when the cursor is moved to it and the 
selection button clicked, provides the data associated with 
the marked audio object which is most recent. 

Having thus described our invention, what we claim as 
new and desire to secure by Letters Patent is: 

1. In a communications network having a head end server 
and a plurality of subscribers connected to said head end 
server, a method of customizing responses to selection 
objects in a video program profile, said method comprising: 

a. transmitting selected video programs from said head 
end server to a subscriber or a group of said plurality of 
subscribers in response to requests for said selected 
video programs from said subscriber or said group said 
plurality of subscribers; 

b. each of said subscriber or said group of said plurality 
of subscribers selecting objects in received ones of said 
selected video programs, each of said objects being an 
image in a received one of said selected video 
programs, each of said objects not being a part of any 
supplemental information added to any one of said 
selected video programs, and each of said objects 
corresponding to an embedded data file containing 
information on a subject which corresponds to the 
appearance of each of said objects; 

c. maintaining a selection database of a history of object 
selections history information, wherein subscriber 
selection history information is acquired for each of 
said subscriber or said group of subscribers; and 

d. dynamically adjusting correspondence of said objects 
to embedded data files transmitted along with received 
one of said selected video programs in response to said 
subscriber selection history information, wherein each 
of said subscriber or said group of said plurality of 
subscribers receives customized infonnation in 
response to selecting one of said objects in said 
received one of said selected video programs. 

2. The method of customizing responses to selection 
objects in a video profile as claimed in claim 1, wherein said 
subscriber selection history information is transmitted, at the 
discretion of a viewer, instantly to said head end server with 
granularity of a single channel viewing or object selection, 
and said transmission is unobtrusive to the viewer. 

3. The method of customizing responses to selection 
objects in a video program file as claimed in claim 1, 
wherein a number of said embedded data files are dynami- 
cally changed in accordance with said subscriber selection 
history information, and in accordance with database search 
activity results. 

***** 
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